Attention-Aligned Transformer for Image Captioning

Recently, attention-based image captioning models, which are expected to ground correct regions for proper word generations, have achieved remarkable performance. However, some researchers argued “deviated focus” problem of existing attention mechanisms in determining the effective and influential features. In this paper, we present A2 - an attention-aligned Transformer captioning, guides learning a perturbation-based self-supervised manner, without any annotation overhead. Specifically, add mask operation on through learnable network estimate true function ultimate description generation. We hypothesize that necessary region features, where small disturbance causes obvious performance degradation, deserve more weight. Then, propose four aligned strategies use information refine weight distribution. Under such pattern, attended correctly with output words. Extensive experiments conducted MS COCO dataset demonstrate proposed consistently outperforms baselines both automatic metrics human evaluation. Trained models code reproducing publicly available.